PastVision: Exploring “Seeing” into the Near Past with Thermal Touch Sensing and Object Detection For Robot Monitoring of Medicine Intake by Dementia Patients

We present PastVision, a proof-of-concept approach that explores combining thermal touch sensing and object detection to infer recent actions by a person which have not been directly observed by a system. Inferring such past actions has received little attention yet in the literature, but would be highly useful in scenarios in which sensing can fail (e.g., due to occlusions) and the cost of not recognizing an action is high. In particular, we focus on one such application, involving a robot which should monitor if an elderly person with dementia has taken medicine. For this application, we explore how to combine detection of touches and objects, as well as how heat traces vary based on materials and a person's grip, and how robot motions and activity models can be leveraged. The observed results indicate promise for the proposed approach.